This invention relates to panoramic image sensing of a super wide-angle field of view, and in particular, it relates to such image sensing using a two-mirror subsystem that is substantially self-corrected. The term xe2x80x9cpanoramicxe2x80x9d means a 360xc2x0 field of view in a horizontal plane while the term xe2x80x9csuper wide-anglexe2x80x9d means a 120xc2x0 or larger field of view in a vertical plane. Preferably, the field of view in a vertical plane is greater than about 1800. More preferably, it is greater than about 200xc2x0. And, for the desired apparatus, most preferably, it is greater than about 260xc2x0.
A perspective imaging system collects rays of light from the scene that pass through a single point of reference and projects them onto a sensing element such as film or a charge coupled device (CCD). The single point of reference in the perspective imaging system is known as the viewpoint of the system. Yamazawa et al., xe2x80x9cOmnidirectional Imaging with Hyperboloidal Projectionxe2x80x9d, IEEE International Conference on Robotics and Automation, 1993, by Nalwa, xe2x80x9cA True Omnidirectional Viewerxe2x80x9d, ATT Bell Laboratories Technical Memorandum, BL0115500-960115-01, January 1996 and by Nayar, xe2x80x9cOmnidirectional Video Cameraxe2x80x9d, DARPA Image Understanding Workshop, May 1997, all incorporated herein by reference, describe the need for a single viewpoint. We have determined that the nature of light propagation through the imaging system and the shape of imaging sensor may introduce geometric transformations in the image projected onto the sensing element. In a large number of applications including surveillance, remote sensing, navigation, model acquisition, virtual reality, computer vision and robotics, it is desirable that these geometric transformations be corrected for the purposes of viewing and analysis. The lack of a single viewpoint introduces aberrations in pupils which manifests itself as uncorrectable geometric transformations (distortions).
A classification of imaging systems based on their field of view is:
1. Traditional imaging systems that image a narrow field of view, usually an octant of the sphere of view (up to 90 degrees).
2. Panoramic imaging systems that image a panorama of the scene. The field of view can be looked upon as a sphere truncated by two parallel planes giving a 360 degree field of view in the horizontal and a limited field of view in the vertical.
3. Omnidirectional imaging systems that image substantially spherical or substantially hemispherical fields of view.
A classification of the same based on their optical components is:
1. Dioptric systems that use only refractive elements;
2. Catoptric systems that use only reflective elements; and
3. Catadioptric systems that use a combination of reflective and refractive elements.
Perhaps the simplest system that provides perspective projection is a pinhole camera. Traditionally, lenses have been used in place of a pinhole simply because of their superior light gathering ability. But a lens, however wide angle it may be, is limited to a hemispherical field of view while still maintaining a single viewpoint, although it is physically challenging to design such wide-angle lenses.
Lenses that deviate from maintaining a single viewpoint have been described by E. H. Hall et al., xe2x80x9cOmnidirectional Viewing using a Fish Eye Lensxe2x80x9d, SPIE Vol. 728 Optics, Illumination and Image Sensing for Machine Vision, 1986, pp. 250, incorporated herein by reference. Such lenses have been known to achieve larger than hemispherical fields of view, up to 280 degrees in the vertical plane. However, these so-called fish eye lenses are significantly larger and more complex than conventional lenses, and suffer from severe geometric distortions and loss of resolution in the image over the field of view. Moreover, the lack of a single reference point for the rays of light imaged by the lens disqualifies their usage in a large number of applications, described above. U.S. Pat. No. 5,185,667 to Zimmerman and U.S. Pat. No. 5,359,363 to Kuban are descriptions of additional uses of fish eye lenses, all incorporated herein by reference. Thus, of the known dioptric systems, those that seek to preserve a single viewpoint are limited to a narrow field of view.
Using only reflective elements, catoptric imaging systems are the closest to ideal imaging systems. The lack of refractive elements removes the possibility of chromatic aberrations allowing these systems to operate under a wide range of illumination wavelengths. But the greatest advantage of catoptric systems is that the reflective elements can be matched to correct for almost all aberrations that plague imaging systems, including field curvature and spherical aberration in pupils. A disadvantage of such systems is their light gathering ability which requires them to operate with lower F-numbers.
Catadioptric systems have been known to span the entire gamut in terms of field of view, from omnidirectional to panoramic to traditional narrow fields of view. The simplest wide-angle catadioptric system comprises two optical components: a curved non-planar primary reflector placed in front of a objective lens. The complete class of mirror lens combinations that capture wide-angle views while maintaining a single viewpoint has been described by Nayar et al., xe2x80x9cCatadioptric Image Formationxe2x80x9d, IEEE International Conference on Computer Vision, January 1998, incorporated herein by reference. Further, for a mirror to have a single viewpoint it is necessary that the mirror be a surface of revolution of a two dimensional curve. Daniel Drucker et al., xe2x80x9cA Natural Classification of Curves and Surfaces With Reflection Propertiesxe2x80x9d, Mathematics Magazine, vol, 69, no. 4, pp. 249-256, 1996, incorporated herein by reference, have shown that the only two dimensional curves with focal properties are conic sections. Hence, the only mirrors that maintain a single viewpoint are conic sections of revolution. Further, those that can be realized in practice are paraboloids, hyperboloids and ellipsoids. It is worthwhile mentioning here that while the sphere is an intuitive solution it is impractical because the focus is at the center of the sphere, and so is the cone for the reason that the focus is at the apex of the cone.
A catadioptric omnidirectional imaging system has been described in U.S. Pat. No. 5,760,826 to Shree Nayar, incorporated herein by reference. The system uses a convex paraboloidal mirror, telecentric relay objective lens and a standard camera lens which projects an annular image of a substantially hemispherical scene captured from a single viewpoint onto a planar sensing device such as a CCD. A disadvantage of the system is that the use of an aspheric surface results in residual field curvature. This prevents its usage with a low F-number compact system.
A more complex catadioptric panoramic imaging system is one that uses two reflecting surfaces in conjunction with a relay objective lens. In such a system the primary reflector collects scene intensity information which is then reflected off a secondary reflector into the relay objective lens.
For the entire system to have a single viewpoint, while the primary mirror must have a single viewpoint (which is the viewpoint of the overall system) it is not necessary for the secondary mirror to have a single viewpoint. The tools for developing such systems of mirrors that have an overall single viewpoint have been described by Conbleet, xe2x80x9cMicrowave and Optical Ray Geometryxe2x80x9d, Published by John Wiley and Sons, 1984, incorporated herein by reference. It can be shown that a variety of mirror pairs, some with exotic shapes, can be used to construct catadioptric imaging systems of interest. However we have determined that mirrors with complex shapes produce confounding optical aberrations. Moreover, even seemingly simple surfaces such as quadrics (surfaces of revolution of planar 2nd order algebraic curves) can produce complex optical aberrations. In our investigation we have found that the only quadrics that can form optically acceptable two mirror systems are conic sections of revolutions, viz. paraboloids, ellipsoids and hyperboloids.
It is a well-known fact in geometry that, a property of conic sections (and hence mirrors formed from conic sections of revolution) is that rays of light in the direction of the one focus of a conic section are reflected by the conic to converge at the other focus of the conic section. Hence, we have determined that for a conic mirror pair to maintain a single overall viewpoint it is necessary the two conics be confocal, i.e., the far focus of the primary conic mirror coincides with the near focus of the secondary conic mirror (a special case is the parabola, where the common focus is at infinity). When the two mirrors are confocal, i.e., the system maintains a single reference viewpoint, the two mirror subsystem corrects for spherical aberration in pupils. The imaging system is made complete by positioning the entrance pupil of the relay objective lens at the far focus of the secondary conic.
Two-mirror panoramic imaging systems have been described at an internet web site maintained by Jeffrey Charles and in U.S. Pat. No. 4,484,801 to Cox. While these systems are panoramic in nature, none seek to preserve a single reference viewpoint, resulting in severe geometric distortion in the image captured off the primary reflector. This distortion manifests itself in the form of spherical aberration in pupils. The complexity of these systems lies in their relay objective lenses, which are required to compensate for such severe spherical aberrations. These systems also exhibit complex field curvature, by far the most difficult aberration to correct. As a result, these systems have high F-numbers resulting in additional complexity in the relay objective lenses. Jeffrey Charles"" system has an F-number that ranges from 22 to 16. Cox""s system contains a 12-component lens for field curvature correction.
Another example of similar panoramic systems is described in International Patent Application PCT/US97/09313 by Driscoll, et. al. The system uses a primary convex paraboloid reflector and a secondary planar reflector and hence exhibits severe astigmatism and field curvature, requiring additional astigmatism correction lenses and field flattening lenses. Additionally, the use of strong elements in the astigmatism correction lens introduces deleterious amounts of spherical aberration (lack of a single reference viewpoint) and coma into the final image.
Another example of a two-mirror catadioptric system is described in U.S. Pat. No. 3,505,465 to Rees. The system uses a convex hyperboloidal mirror as the primary reflector and a convex spherical mirror as the secondary reflector in conjunction with a complex relay objective lens. The use of two convex mirrors causes the system to exhibit severe field curvature, hence the need for a complex relay objective lens.
Another example of a two-mirror system is described in U.S. Pat. No. 4,566,763 to Greguss that uses two paraboloidal reflective surfaces, a refractive surface and a telecentric objective lens. It can be shown that such a confocal mirror subsystem with two paraboloids that seeks to preserve a single overall viewpoint must use a perspective objective relay lens, and hence the system described there does not preserve a single reference viewpoint.
U.S. Pat. No. 4,395,093 to Rosendahl and Dicks describes a two mirror system in which the primary reflector is a convex hyperboloid and the secondary reflector is a concave hyperboloid in conjunction with a complex objective lens that comprises 21 components. Since the two reflectors have very different radii of curvatures, the system exhibits severe field curvature, which is corrected in part by the complex objective lens.
U.S. Pat. No. 5,631,778 to Ian Powell describes a panoramic imaging system with two reflectors and a complex refractive objective lens. The primary reflector is a concave conic of revolution: either an ellipsoid, sphere or oblate spheroid. The secondary reflector is a convex conic of revolution, typically a hyperboloid although spheres have been used too. Since the primary mirror is concave, the field of view in the vertical plane is limited to 180xc2x0. An additional 10xc2x0 is possible by adding a refractive negative shell in front of the entire arrangement. Investigation has shown that no effort was apparently made to make the two mirrors confocal and further, it is evident that the use of a sphere or oblate spheroid in conjunction with a hyperboloid, results in severe spherical aberration of pupils. Hence the system does not have a single viewpoint, indicating the need for a complex relay objective lens.
U.S. Pat. No. 5,627,675 to Davis et al. discloses a panoramic imaging system that employs as its primary collection subsystem two elements that comprise a Mersenne optic; the first element being a convex paraboloidal reflector and the second element being a concave paraboloidal reflector. From the above discussion, such a system of paraboloidal mirrors comprises a confocal pair of conic section mirrors which preserves a single reference viewpoint, thus eliminating to a large degree aberration in pupils. However, for any field correction to occur in the two mirror subsystem, the two mirrors must have the same radii of curvature. Doing so severely restricts the field of view due to vignetting of the scene by the secondary reflector, typically to less than 45 degrees above and below the horizon.
To summarize, the catadioptric panoramic imaging systems known in the art all have shortcomings. Most systems do not have a single reference viewpoint for the imaging system. This results in severe spherical aberration in pupils that manifests itself as uncorrectable geometric distortion in the captured image. Those systems that do maintain a single viewpoint are limited either in their ability to perform under varying light conditions and F-numbers due to significant field curvature that remains uncorrected, or in their vertical field of view due to vignetting by optical components. Further, the use of objective lenses for correcting above mentioned spherical aberration and field curvature results in chromatic aberrations. Furthermore, these objective lenses tend to be highly complex and expensive.
The shortcomings of the above-mentioned state of the art are substantially remedied by the invention disclosed here. The patents and publications referenced above do not teach the panoramic imaging apparatus and method disclosed here.
One aspect of the invention is a panoramic imaging apparatus with a super wide field of view for imaging a scene, comprising:
a. an image sensor positioned to receive said super wide field of view;
b. a two reflector sub-system that is substantially self corrected, said subsystem comprising:
i. a primary truncated reflector positioned to reflect an image of said substantially super wide-angle scene,
ii. a secondary truncated reflector optically coupled with said primary reflector, said secondary truncated reflector positioned to reflect said image reflected by said primary reflector;
wherein said primary and secondary reflectors have shapes and sizes to substantially correct field curvature of the image projected on said image sensor; and
c. a relay system, optically coupled to said secondary reflector, for substantially filtering out principle rays that are not reflected by said secondary reflector.
In an exemplary embodiment, the two mirrors have shapes and sizes to capture the super wide-angle scene from a single reference viewpoint.
A super wide-angle field of view, for the purpose of this invention, comprises a field of view greater than or equal to 120 degrees. That is, it constitutes a cone which extends at least 60 degrees from the optical axis. There may exist a blind spot in the immediate vicinity of the optical axis. Preferably, the super wide-angle field of view of the present invention includes a panoramic field of view greater than or equal to 180 degrees, more preferably greater than or equal to 200 degrees. In selected embodiments of the invention, the field of view may be greater than or equal to 220 degrees.
In an exemplary embodiment of an apparatus according to the present invention, the primary reflector is convex. In another exemplary arrangement, the primary reflector is concave.
In yet another exemplary embodiment, the surface of the primary reflector is a conic of revolution that obeys the following equation,
r2=2R1zxe2x88x92(1+k1)z2.xe2x80x83xe2x80x83(1)
Since the reflector has axial symmetry, equation (1) also represents a cross section of the reflector r is the radial coordinate, i.e., r2=x2+y2, and z is the coordinate along the optical axis Z. R1 is the radius of curvature of the conic that forms the reflector and k1 is its conic constant. For k1 less than xe2x88x921, the reflector is a hyperboloid of revolution, for k1=xe2x88x921, a paraboloid, for xe2x88x921 less than k1xe2x89xa60. Note that for the special case of k1=0, the primary reflector is a sphere.
The reflector is truncated in a plane that is perpendicular to the optical axis, Z, such that the desired field of view is imaged by the reflector.
A hole, having a selected diameter, is created at the vertex (apex) of the reflector to create an optical path through which light reflected off the secondary reflector can pass without substantial vignetting. The hole serves the additional purpose of discarding rays of light that are reflected by the primary reflector such that they do not substantially pass through the single viewpoint of the imaging system.
In another exemplary arrangement, the primary reflector is a higher order ( greater than 2, e.g., 14) surface of revolution.
In an exemplary embodiment of an apparatus according to the present invention, the secondary reflector is concave. In an alternative arrangement, the secondary reflector is convex.
In yet another exemplary embodiment, the secondary reflector is in the form of a conic of revolution. The secondary reflector is preferably positioned along an optical axis, which coincides with the optical axis, Z, of the primary reflector. The surface of the secondary reflector obeys the following equation,
r2=2R2zxe2x88x92(1+k2)z2,xe2x80x83xe2x80x83(2)
where once again, due to axial symmetry, equation (2) also represents a cross section of the reflector wherein r is the radial coordinate and z is the coordinate along the optical axis, Z. R2 is the radius of curvature of the conic that forms the secondary reflector and k2 is its conic constant. For k2 less than xe2x88x921, the reflector is a hyperboloid of revolution, for k2=xe2x88x921, a paraboloid, for xe2x88x92 less than k2 less than 0, an ellipsoid. Note that for the special case of k2=0, the secondary reflector is a sphere.
The reflector is truncated in a plane that is perpendicular to the optical axis, Z, such that light reflected off the primary reflector is imaged by the secondary reflector.
In another exemplary arrangement, the secondary reflector is a higher order ( greater than 2, e.g., 14) surface of revolution.
In an exemplary embodiment, the distance between the apexes of the two reflectors substantially obeys the following equation:                               d          1                =                                            2              ⁢                              R                1                            ⁢                                                -                                      k                    1                                                                                      "LeftBracketingBar"                              1                +                                  k                  1                                            "RightBracketingBar"                                +                                    R              2                                      1              +                                                -                                      k                    2                                                                                -                                    R                              1                +                                                      -                                          k                      1                                                                                            .                                              (        3        )            
In an exemplary embodiment, the relay system is preferably positioned along an optical axis, which coincides with the common optical axis, Z, of the primary and secondary reflectors, such that the primary reflector is located physically between the relay system and the secondary reflector. The relay system is an optical component used to project an image to another location, i.e., it is a means for relaying the image to the sensor.
In another exemplary arrangement, the relay system is a simple aperture, as in a pin-hole camera.
In another exemplary arrangement, the relay system comprises at least one lens. The lens may be made of a plurality of portions of optical material that are cemented together, such as a doublet.
In an exemplary embodiment, the distance between the vertex of the primary reflector and the entrance pupil of the relay system substantially obeys the equation:                               d          2                =                                            2              ⁢                              R                2                            ⁢                                                -                                      k                    2                                                                                      "LeftBracketingBar"                              1                +                                  k                  2                                            "RightBracketingBar"                                +                                    R              1                                      1              +                                                -                                      k                    1                                                                                -                                                    2                ⁢                                  R                  1                                ⁢                                  xe2x80x83                                ⁢                                                      -                                          k                      1                                                                                                  "LeftBracketingBar"                                  1                  +                                      k                    1                                                  "RightBracketingBar"                                      .                                              (        4        )            
In an exemplary arrangement, the image sensor is electronic, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor sensor (CMOS) and provides an electronic signal that is representative of the image projected onto the image sensor. This image signal is digitized and transferred to an image storage apparatus. The digitized image can then be transferred to an image processing apparatus. In another exemplary arrangement, the digitized signal is transferred directly to an image processing apparatus, without using an intermediate storage apparatus. In yet another exemplary arrangement, the image sensor is photographic film, the image of which can be subsequently digitized and the resulting signal then transferred to the imaging processing apparatus. The imaging processing apparatus is advantageously adapted to enable viewing of any portion of the super wide-angle scene.
Another aspect of the present invention is a two mirror system that substantially eliminates to a very large extent, aberrations that occur in panoramic optical systems, allowing for the creation of an image of high optical quality, said system comprising:
a. a primary reflector that constitutes the first mirror, and
b. a secondary reflector that constitutes the second mirror, the shape and size of which is carefully matched to the shape and size of said primary reflector.
To minimize field curvature in an optical system, the Petzval curvature of the system must be close to zero. While this corrects third order field curvature, higher orders of the field curvature can be corrected by optimizing optical powers of the optical components. The result is a diffraction-limited system with highest possible image quality.
Yet another aspect of the present invention is a two mirror subsystem that substantially reduces the cost of manufacturing a commercial super wide-angle panoramic imaging apparatus, and increases versatility of use, said subsystem comprising:
a. a primary reflector that constitutes the first mirror, and
b. a secondary reflector that constitutes the second mirror, the shape and size of the two mirrors chosen to minimize field curvature introduced by the two mirror subsystem.
It is known that in systems with field curvature, a non-trivial array of lenses is required to offset the negative effects of field curvature, this array increasing the cost of the system. Additionally, field curvature correcting lenses or field flatteners are typically introduced in close proximity to the image sensor, reducing the versatility of the system with respect to use with standard, off-the-shelf components.
Another aspect of the present invention is a panoramic imaging apparatus that senses a super wide-angle scene from a single reference viewpoint. In an exemplary embodiment, the reference viewpoint is a locus of points that lie within a sphere of radius of no more than 4 mm.
The present invention also provides a method for sensing an image of super wide-angle scene, which in an exemplary embodiment of the present invention, comprises the following steps:
a) reflecting an image of the super wide-angle field of view on a primary reflector (described above) such that the single viewpoint substantially coincides with the near focus of the conic section that forms the primary reflector,
b) reflecting the image reflected in step (a) on a secondary reflector (described above) such that the rays of light reflected by the secondary reflector pass through the hole on the primary reflector,
c) propagating the rays of light from step (b) through a relay system and
d) sensing the rays of light propagated through the relay system in step (c).
The present invention also includes, in another exemplary embodiment of the imaging method, the further steps of:
1. providing an image signal which is representative of the image projected onto the image sensor,
2. converting the image signal to image data, mapping the image data into a Cartesian-coordinate system, and
3. interpolating the image data and forming a digital image from the mapped image data and the interpolated image data.
Another aspect of the present invention is a two mirror panoramic system that is substantially scalable, in that the size of all components and their relative positions can be scaled without significant loss of image quality.